-
-
Notifications
You must be signed in to change notification settings - Fork 14.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
libiconvReal: implement ABI compatibility on Darwin #238993
Conversation
I see there isn’t a maintainer. Should I add myself? Since I want to move Darwin to using upstream where possible, it’s not ideal to depend on an unmaintained package (even if it works and doesn’t change often). |
Result of 1 package built:
|
Result of 1 package built:
|
Result of 1 package built:
|
Result of 1 package built:
|
# | ||
# For an explanation why `libcharset.dylib` is reexported, see: | ||
# https://github.com/apple-oss-distributions/libiconv/blob/a167071feb7a83a01b27ec8d238590c14eb6faff/xcodeconfig/libiconv.xcconfig | ||
postBuild = lib.optionalString stdenv.targetPlatform.isDarwin '' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you sure targetPlatform
is right here, not hostPlatform
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to take this opportunity to ask the same question regarding the override here:
nixpkgs/pkgs/os-specific/darwin/apple-sdk-11.0/default.nix
Lines 61 to 69 in d6b7d2b
mkStdenv = stdenv: | |
if stdenv.isAarch64 then stdenv | |
else | |
(overrideCC stdenv (mkCc stdenv.cc)).override { | |
targetPlatform = stdenv.targetPlatform // { | |
darwinMinVersion = "10.12"; | |
darwinSdkVersion = "11.0"; | |
}; | |
}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both should be hostPlatform
. I’ll submit a separate PR to fix the 11.0 stdenv definition and push an update to fix this one. Thanks for the feedback both of you.
libiconvReal after this commit: $ nix build .#libiconvReal
$ otool -L ./result/lib/libiconv.2.dylib
./result/lib/libiconv.2.dylib:
/nix/store/i7xdmh2v362nn88jnqgzpnvpx8qfjs8b-libiconv-1.16/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
/nix/store/i7xdmh2v362nn88jnqgzpnvpx8qfjs8b-libiconv-1.16/lib/libcharset.1.dylib (compatibility version 2.0.0, current version 2.0.0, reexport)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
$ nm ./result/lib/libiconv.2.dylib | rg iconv
00000000000e4088 D __libiconv_version
000000000000437c T _iconv
0000000000004ae4 T _iconv_canonicalize
00000000000043a0 T _iconv_close
0000000000003374 T _iconv_open
00000000000043b8 T _iconv_open_into
0000000000004750 T _iconvctl
00000000000048a0 T _iconvlist
0000000000017b5c t _libiconv_relocate
0000000000017c40 t _libiconv_relocate2
0000000000017a94 T _libiconv_set_relocation_prefix This is libiconv extracted from the dyld cache on a macOS 13.4 system. $ otool -L libiconv.2.dylib
libiconv.2.dylib:
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/usr/lib/libcharset.1.dylib (compatibility version 2.0.0, current version 2.0.0, reexport)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
$ nm libiconv.2.dylib
00000001d8ce15c0 D __libiconv_version
000000018bdc80e0 T _iconv
000000018bddba50 s _iconv_2VersionNumber
000000018bddba20 s _iconv_2VersionString
000000018bdc8558 T _iconv_canonicalize
000000018bdc8104 T _iconv_close
000000018bdc7064 T _iconv_open
000000018bdc8120 T _iconvctl
000000018bdc8284 T _iconvlist
000000018bddb89c T _libiconv_set_relocation_prefix I believe the The version differences appear to be an artifact of how libtool implements versioning on Darwin. Apple’s libiconv s 6:0:4 while 1.16 is 8:1:6, which should be compatible according to the libtool documentation. |
substituteInPlace "include/$iconv_h_in" \ | ||
--replace "#define iconv libiconv" "" \ | ||
--replace "#define iconv_close libiconv_close" "" \ | ||
--replace "#define iconv_open libiconv_open" "" \ | ||
--replace "#define iconv_open_into libiconv_open_into" "" \ | ||
--replace "#define iconvctl libiconvctl" "" \ | ||
--replace "#define iconvlist libiconvlist" "" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if it might not be a good idea to export both versions of symbols to increase compatibility with the Linux version?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both glibc and Musl use the iconv
prefix. The reason for the libiconv
prefix in GNU libiconv is to allow it to be used on systems that provide their own implementations in libc. However, on Darwin, the system implementation is GNU libiconv, and it’s being (or will be) replaced with this version.
As far as I can tell, (almost?) all Linux packages in nixpkgs use iconv
from libc (glibc or Musl). It’s possible a Darwin app might link both the system and the separate libiconv, but I’d like to see an actual example before attempting to support that use case. I’d had to add to Darwin’s maintenance load to support something no one actually does.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aha, makes sense. I just worry about having the same pkgs.libiconvReal
with different behaviour across platforms, even though it's probably not going to come up. How would you feel about adding a forDarwinStdenv
(or whatever) option to the package to conditionalize this logic on, and setting that in the pkgs.libiconv
logic rather than changing pkgs.libiconvReal
directly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe call it enableDarwinABICompat
and default it to stdenv.hostplatform.isDarwin
? If a package wants to do something wacky, it can override the compatibility and use both. I’ve pushed an updated branch with that change. Let me know what you think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My suggestion is to call it e.g. useLibcAbi
, default it to false
, and then adjust all-packages.nix
in this manner:
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index cca76287015..69426ff28d1 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -22372,9 +22372,7 @@ with pkgs;
then libcIconv (if stdenv.hostPlatform != stdenv.buildPlatform
then libcCross
else stdenv.cc.libc)
- else if stdenv.hostPlatform.isDarwin
- then darwin.libiconv
- else libiconvReal;
+ else libiconvReal.override { useLibcAbi = true; };
libcIconv = libc: let
inherit (libc) pname version;
Advantages: one fewer Darwin-specific conditional, might fix issues on other platforms that currently use libiconvReal
here with probably the wrong ABI for the use-case, keeps the comment above it accurate ("We also provide libiconvReal
, which will always be a standalone libiconv, just in case you want it regardless of platform." – I think if we provide something called libiconvReal
with this documentation it should match the API/ABI of the actual standalone libiconv, so if we do something other than this I think we need to change the name and comment. But it seems fine to me to just do the override in libiconv
's definition, because it expresses what we want from it.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As discussed on Matrix, I’d like to keep the change conservative and targeted at Darwin for now. Other platforms may have their own iconv implementation, so it would be a misnomer to call that the “libc ABI”. I also don’t want to propagate Darwin’s quirk in reexporting libcharset
. At least for now, I’ll be keeping the name but defaulting it to false
.
When it comes time to switch Darwin to this implementation, the definition of libiconv
can be updated to libiconvReal.override { enableDarwinABICompat = true; }
. That will require updates to a few other packages that reference darwin.libiconv directly and the implementation of a deprecation mechanism for the darwin
attrset.
Thanks for the feedback! I’ve pushed a commit that defaults this to off.
71adcb1
to
bb34ffe
Compare
This commit prepares libiconvReal to replace darwin.libiconv, allowing it to be used with binary derivations that patch out references to the system libiconv with one from nixpkgs. Apple’s libiconv is based on GNU libiconv 1.11 (the last version before it switched to LGPLv3+). Any newer releases by Apple appear to be build system tweaks. The core sources are barely updated. This means that Darwin users won’t get any fixes from upstream updates, and maintaining darwin.libiconv requires dealing with a separate and different build system (because Apple now builds it with Xcode). Fortunately, it is possible to build upstream libiconv in a way that is compatible with Apple’s distribution of it. There are two things that need to happen to produce an ABI-compatible build of libiconv: * Existing symbols need to be exported with the `iconv_` prefix instead of the `libiconv_` prefix. New symbols can have the `libiconv` prefix, and one symbol in Apple’s distribution does, but older ones must have the older prefix; and * Reexport `libcharset.dylib` from `libiconv.dylib`. This is explained by Apple as the result of a bug in their transition to an Xcode-based build system. Both these these are doable and have been done by this commit. I have tested it with building GHC, which downloads a binary distribution as part of its bootstrap and replaces references to the system libiconv with darwin.libiconv. Using this patch, libiconvReal is able to work without any changes to the GHC derivation. Note that this patch does not actually deprecate or remove darwin.libiconv yet. That will be done in a future patch after Darwin support is added for aliases and deprecating packages in the `darwin` attrset.
bb34ffe
to
057dd0e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Things done
This commit prepares libiconvReal to replace darwin.libiconv, allowing it to be used with binary derivations that patch out references to the system libiconv with one from nixpkgs.
Apple’s libiconv is based on GNU libiconv 1.11 (the last version before it switched to LGPLv3+). Any newer releases by Apple appear to be build system tweaks. The core sources are barely updated. This means that Darwin users won’t get any fixes from upstream updates, and maintaining darwin.libiconv requires dealing with a separate and different build system (because Apple now builds it with Xcode). Fortunately, it is possible to build upstream libiconv in a way that is compatible with Apple’s distribution of it.
There are two things that need to happen to produce an ABI-compatible build of libiconv:
iconv_
prefix instead of thelibiconv_
prefix. New symbols can have thelibiconv
prefix, and one symbol in Apple’s distribution does, but older ones must have the older prefix; andlibcharset.dylib
fromlibiconv.dylib
. This is explained by Apple as the result of a bug in their transition to an Xcode-based build system.Both these these are doable and have been done by this commit. I have tested it with building GHC, which downloads a binary distribution as part of its bootstrap and replaces references to the system libiconv with darwin.libiconv. Using this patch, libiconvReal is able to work without any changes to the GHC derivation.
Note that this patch does not actually deprecate or remove darwin.libiconv yet. That will be done in a future patch after Darwin support is added for aliases and deprecating packages in the
darwin
attrset.Description of changes
sandbox = true
set innix.conf
? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"
. Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/
)